Totally Model-Free Reinforcement Learning by Actor-Critic Elman Networks in Non-Markovian Domains

In this paper we describe how an actor critic rein forcement learning agent in a non Markovian domain nds an optimal sequence of actions in a totally model free fashion that is the agent neither learns transitional probabilities and associated rewards nor by how much the state space should be augmented so that the Markov prop erty holds In particular we employ an Elman type re current neural network to solve non Markovian problems since an Elman type network is able to implicitly and automatically render the process Markovian A standard actor critic neural network model has two separate components the action actor network and the value critic network In animal brains however those two presumably may not be distinct but rather somehow entwined We thus construct one Elman network with two output nodes actor node and critic node and a portion of the shared hidden layer is fed back as the context layer which functions as a history memory to produce sensitiv ity to non Markovian dependencies The agent explores small scale three and four stage tri angular path networks to learn an optimal sequence of ac tions that maximizes total value or reward associated with its transition from vertex to vertex The posed prob lem has deterministic transition and reward associated with each allowable action although either could be stochastic and is rendered non Markovian by the reward being depen dent on an earlier transition Due to the nature of neural model free learning the agent needs many iterations to nd the optimal actions even in small scale path problems